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ABSTRACT 

This paper presents the results of a study on the 
effects of using 3 simulators with 14 computer science majors 
enrolled in an undergraduate computer architecture course. Three 
ability groups (high, medium, and low) were formed based on the 
students* quality point averages for prior computer science courses. 
Analyses of data from a pretest, a posttest, instructor-prepared 
achievement tests following each of the simulators, programming 
assignments, and survey results showed: (1) no significant 
differences among the three groups on the pretest and posttest 
scores; (2) only the low and medium ability groups showed significant 
performance gains from pretest to posttest; (3) the use of the 
simulators equalized the student performance in the three ability 
groups; (4) nonsignificant differences on both programming 
assignments for all ability groups; (5) positive student attitudes 
toward and perceptions of the use of simulators; and (6) student 
perceptions that the simulators made learning about computer 
architecture more concrete, and helped in writing and understanding 
programs and microsequences of instructions. Suggestions for further 
research are presented including using larger samples, different 
methodology for defining the ability groups, and the use of 
qualitative interviews to ascertain student perceptions. Six figures 
present the results of the various statistical analyses. (Contains 5 
references . ) (ALF) 
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ABSTRACT 



Research indicates that computer simulations in higher education settings 
generally result in better or equivalent student performance. This paper presents the 
results of a study on the effects of using three simulators in an xmdergraduate computer 
architecture course. 

Three ability groups were formed based on the subjects' quality point averages 
(QPA) for prior computer science courses. Analyses of the pretest and posttest showed 
no significant differences among the three groups. Overall, only the low and medium 
ability groups showed significant performance gains from pretest to posttest This 
research indicates that the use of the simulators equalized student performance in the 
three ability groups. Additionally, student attitudes and perceptions about the use of 
the simulators were positive. 



OBJECTIVES 

Having a course which deviates from the students' norm makes for difficulty in 
learning. The unconventional and abstract nature of the computer architecture course 
material needs to be simplified for the students' comprehension without reducing the 
quality of the course. Clariana (1988-1989), Ltmetta and Hofstein (1981), and McGuire 
(1976) suggest that providing students with computer-simulated learning materials will 
allow them to have a realistic xmderstanding of abstract material. 

Realistically, students caimot study the internal architecture of a computer system. 
However, computer programs to simulate the architecture are used frequently when this 
subject is studied. In similar situations in which actual and simulated science 
experiments were studied, Boblick (1972) and Cavin and Lagowski (1978) showed that 
students using computer simulations performed as well as or better than students who 
performed the actual experiments. The incorporation of computer simulated learning 
materials may maintain the integrity of the course content while decreasing the level of 
:x>mprehension complexity. 

This research was motivated by the need for supportive learning materials for the 
students in a computer architecture course. With the support of interactive computer 
learning aids, the researchers found that students had a better imderstanding of the 
course material. 

Several research questions were addressed in this study: 

1) To what extent do the use of computer-based simulators affec^ the students 
in low, medium, and high ability groups? 
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2) To what extent do the use of computer-based simulators affect student 
achievement in the computer architecture course? 

3) How do students perceive the use of simulators in a computer architecture 
course? 

DESIGN OF THE STUDY 

Sample 

Fourteen undergraduate students enrolled in a computer architecture coxxrse 
served as the subjects in this study. The students were computer science majors in a 
small state imiversity in southwestern Pennsylvania. 

Independent Variables 

Blocking Variable. The students were separated into low, medium, and high ability 
groups based on their quality point average (QPA) for computer science courses taken 
prior to the study. The average QPA score for all students in the study was calculated. 
Those scoring at least one standard deviation above the mean were placed in the high- 
ability group. Those scoring at least one standard deviation below the mean were 
placed in the low-ability group. The remaining students were placed in the medium- 
ability group. 

Treatment Students from all ability groups began the semester with an 
introduction to the first computer architecture simulator. This simulator provided the 
students with an introductory instructional tutorial of a hypotheticcd computer 
architecture. Additionally, this first package included a content quiz followed by an 
interactive machine-code program simulator. The package presented the first 11 
instructions of the hypothetical computer desaibed in the textbook used in the course. 
The machine-code simulator demonstrated "canned" as well as student-written programs. 

Near the midpoint of the course, the stude its used a second simulator 
demonstrating data flow at the machine level. This simulator was used to support the 
class lectures on microsequences; however, the students had the opportunity to operate 
this simulator outside the class. 

Near the end of the semester, the students used a simulator containing the entire 
instruction set of the hyf>othetical computer. This simulator accepted programs written 
by the students and demonstrated the execution sequence for a program. Several 
programs were assigned to be written using this third simulator. 

Dependent Measures 

Pretest. At the begirming of the semester, the students were administered a 
pretest designed by the course instructor. A panel of experts validated the pretest. The 
results of the pretest were used to establish the students' initial level of computer 
architecture knowledge. Significant differences in achievement between the three groups 
were investigated. 



Achievement Tests, The students were administered instructor-prepared tests after 
each of the simulators was used* 

Programming Assignments. The students wrote and demonstrated two programs 
using the third sxm\alator. 

Posttest. The students were administered the posttest at the end of the course. 

Surveys. The subjects completed a 13-item survey which exammed the time they 
spent on eadi of the three simulators and their perceptions of the effects the simulators 
had on their imderstanding of computer architecture concepts, the instruction set of the 
computer, the effects of the simulators on their abilities to write programs, and on the 
overall effectiveness of the simulators. 

RESULTS 

A one-way ANOVA was used to compare the meems of the pretest, examination 1, 
examination 2, and posttest scores for the subjects in the three ability groups. The 
results are represented in a chart (see Figure 1). 

Significant differences were found among the groups on examination 1 and 
examination 2. To determine the source of the significant differences, the Scheff^'s test 
was used. On examination 1, significant differences were found between the Low and 
Medium ability group means and between the Low and High ability group means. The 
Low ability group performed significantly lower than the Medium or High ability 
groups. On exan:unation 2, significant differences were foxmd between the Low and 
High ability group means, favoring the High ability group. 

T-tests were performed to determine whether or not significant gains were made 
between the various examinations. Those results can be found in Figure 2. Significant 
differences were foimd between the pretest and examination one, examination two and 
the posttest, and the pretest and posttest scores. 

T-tests were used to compare the gains for each of the groups. The results can 
be found in Figure 3, Figure 4, and Figure 5. For the Low ability group, there were 
significant gains pretest to examination one and pretest to posttest. For the Medium 
ability group, there were significant gains pretest to examination one, examination two 
to posttest, and pretest to posttest. There were nonsignificant gains for the High ability 
group. 

A one-way ANOVA was used to compare the means on the two programming 
assignments for the subjects in the three ability groups. There were nonsignificant 
differences on both assignments. 

Percentage changes were calculated for the three ability groups between the 
various examinations. These gains can be foxmd in Figure 6. The Medium ability group 
made the greatest gains from pretest to examination one. The Low ability group made 
the greatest gains between examination one and examination two and between 
examination two and the posttest. The Medium ability group made the greatest percent 
gains between pretest to posttest. 
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DISCUSSION 



The ANOVA performed on tiie pretest means for the three ability groups (F=l-52) 
showed no significant differences among those means* This covild be expect^ since the 
content examined on the pretest consisted of information the students should not have 
known* Therefore, the students were relatively equal in their knowledge of computer 
architecture concepts at the beginning of the semester-long study* 

The significant gains in performance for all students (t=8*25) from pretest to 
examination one were expected* Also, the inaeases for the Low {t=2,97) and Medium 
(t=9*50) groups were significant and were further reflected in the percent gains for those 
two groups (PGl=191,30%, PGi^=319*34%). Although the High ability group outscored 
both the Low and Medium ability groups, the gain for the High ability group (from 
pretest to examination one) was not significant. 

It would be expected that the High ability group would outscore the Low and 
Medium ability groups. What is important is that the gains made by the Low and 
Medium ability groups were significant, while the gedns made by dte High ability group 
were not* Also, the percent gain made by the Low ability group was higher tiian the 
percent gain for the High ability group (PGh=179.75%). Perhaps the use of the first 
simulator helped the Low and Mediimi ability groups to make the significant gains in 
their scores. 

No significant differences were found in the score changes between the first and 
second examinations. However, the results of the ANOVA performed on fhe means of 
the three ability groups on tiie second examination was significant (F=4.95). The 
significant difference was between the Low and High ability group means. While both 
the Medium and High ability groups outscored the Low ability group on the first 
examination, only the High ability group outscored tihe Low ability group on the second 
examination* Each of the three groups increased their means from the first examination 
to the second (PGl=110.68%, PGm=101.75%, PG^=^10270%); however, the greatest 
increase was in the Low ability group. Again, it may be that the use of the first two 
simulators contributed to the reduction in differences between the scores of the students 
in the different ability groups. 

A significant difference was foimd in the test scores between the second 
exanunation and the posttest Overall, the scores dropped. When the ability groups 
were examined, the scores for the Low ability group dropped while those for the 
Medium and High ability groups rose* The gain for the Mediirai ability group was 
statistically significant (t=2.36). Perhaps the drop in overall scores can be explained by 
the nature of the examinations. The second examination tested the information covered 
after the first examination; the posttest was comprehensive. The all-inclusive nature of 
the posttest may explain the drop in scores. 

A one-way ANOVA performed on the scores of the entire group on the posttest 
indicated no significant differences between the three group means (F=0*78), This might 
show that the simulators had a leveling effect on the scores. However, the Low and 
Medium ability groups had significant gains in avera^ scores (tL=4*46, ti^=6.07) while 
the High ability group did not (1^-327). When these gains are expressed as percent 
changes, the Low and Medium ability groups, as expected from the t scores, 
outperformed the High abiUty group (PGl=216.46%, PGm=289.22%, PGh=175.37%). 
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The same comprehensive examination was used as both tiie pretest and the 
posttest. The High ability group made nonsignificant changes throughout. The mean 
on tiKe pretest Sh=54*17) was overshadowed by the means on the other three 
examinations (exam 1: xh=97.37, exam 2: Xh=100.00, posttest: Xh=95.00). There was 
not much room for improvement in tihe scores of the subjects in the HDigh ability group. 

However, there were significant gains in examination scores for the Low and 
Medium ability groups. Subjects in the Low ability group made significant gains pretest 
to examination one and pretest to posttest Those in the Meditrai ability group made 
significant gains pretest to examination one, examination two to posttest, and pretest to 
posttest. We hypothesize tiiat these gains were due, at least in part, to the equalizing 
effect the simulator had on the subjects in the Low and Medium ability groups- 
There were no significant differences between the set of scores on progranuning 
assignment one (xl=%.25, Xj^=85.00, Xfj==95.00, Fprogi=0-26) and on programming 
assignment two ^l=83.75, Xj^=61.25, Xh=95.00, These are not surprising 

results- The students were pwmitted to work on meu* progranuning assignments for a 
specified period of time. However, if the output of tfie assignment was not correct, the 
students were free to alter the code. Perhaps the use of the previous two simulators 
influenced the subjects in the Low and Medium ability groups so that their resulting 
programs were not significantly different from those produced by tiie students in the 
High ability group. 

Another reason for the lack of statistically significant differences in the group 
means on the pretest and posttest may be due to the small sample size (nL=4, n|^==8. 
With a larger sample size, the degrees of freedom would, of course, be greater 
and a smaller calculated F value would have been significant. If the means of the 
groups on the pretest (xl=35.42; x^^=28.02; Xh=54.12) and posttest (xl=76.67; Xj^=81.04; 
Xh=95.00) are examined, it can be seen that they are different from each other; yet, the 
small sample size did not allow for significant differences. 

Each of the 14 students in the study anonymously completed a survey. The 
survey asked for a self-reported indication of the time spent using the three simulators. 
One-half of the students used the first simulator for approximately one and one-half 
hours. Six other students used it more ft-equently. All but one of the students used the 
second simulator for three or fewer hours. Most students used the third simulator for 
two or more hours. The third simulator was probably used for more hours than the first 
two simulators since the students were required to write two programs using that 
simulator. 

For the most part, the students who participated in the study indicated that the 
simulators helped them to understand the concepts relating to the specific machine 
demonstrated in the simulators* They also indicated that the use of the simulators 
helped them to understand the concepts relating to computer architectures* Their 
responses indicated that the simulators made learning about computer architecttu'e more 
concrete. 

Student responses were less strong concerning the use of the simulators to write 
or imderstand programs written for the hypothetical machine. However, the majority 
of the students agreed that the simulators helped in those areas. They also agreed that 
the simulators helped them to understand the instruction set and nucrosequences of 
instructions. 



The students indicated that the simulators helped them better understand how 
other computer structures operate And, although some students felt that they could 
have learned about the architecture of the hypothetical machine without the simulators, 
they recommended its use when the class is next taught. 

CONCLUSIONS 

Students, especially in the Low and Medium ability groups, benefitted from the 
simulators. The use of the simulators may have equalized student achievement in the 
three ability groups. As reported in the surveys, the students had positive attitudes 
about the advantages of using the simulators. 

Although these first findings are encouraging, more research needs to be done in 
this area. Larger sample groups should be studied. Student QPAs are based, in part, 
on the number of courses taken. Since this varies from student to student, the pretest 
should be used to separate tiie subjects into Low, Medium, and High ability groups. We 
feel this would be a better determiner of ability levels at the beginning of the course. 
Also, a more equal distribution of the subjects into the three groups is needed. 

Individual interviews with the subjects should be conducted to gain the subjects' 
perspectives on the use of the simulators. An interview approach would pernut the 
researcher to gain a better perspective on student views of the architecture simulators. 
Additionally, other areas of the computer architecture, such as the ALU, can and should 
be simulated. This is an area of research which deserves further study. 
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Figiire 1 Analysis of Variance for 
Various Exams for the Low^ Medium, and 
High Ability Groups. 
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Figures Results of t-tests for Various 
Exams for the Low Ability Group. 
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Figure 5 Results of t-tests for Various 
Exams for the High Ability Group. 
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Figure 2 Results of t-tests for Various 
Exams for all Subjects. 
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Figure 4 Results of t-tests for Various 
Exams fot the Medium Ability Group. 




Figure 6 Percent Gains of Various Exams 
for the Low, Medium, and High Ability 
Groups 



